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ABSTRACT ; 

( The sustained effects stud i gs^ of_JPi_t le 1 programs in 

130 school systems in Kentucky were analyzed . ^^"pur^ose of the 
review was to. examine the types of questions asked, the results of 
the studies, and the ways^ xn which results were interpreted ?nd 
utilized. There were -u09 summer effects studies. These showed that 
overall, the gains of Title I were sustained over *the summer. There 
were normaL^curve equivalent losses, over the summer but the losses 
may tetlecz no more than the difference between gains made by 
projects on a fall-spring testing cycle and those made by projects on 
a spring-spring cycle. This analysis presents some of the problems 
encountered by local educational agencies (LEA) and suggests 
guidelines to use in the future to improve the studies. The 
significance of the sustained effects studies was not so much in the 
results of the studies as it was in the process of the studies. LEAs 
to\>k a much deeper look at what they were doing in their projects and 
at the consequences of thein efforts. They took a closer look at 
their process of testi'ng and their interpretation of test results in 
the evaluation of program impact. (PN) 
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A META-ANALYSIS OF TITLE I/C&APTER 1 SUSTAINED EFFECTS STUDIES 



The regulations covering Title I (nw Chapter 1) require that the 
evaluations of funded projects include tra determination of whether performance m { 
in the basic skills is sustained over a period of more than one year. To 
determine their gains, many Local Educatioifal Agencies (LEAs) administer nationally 
normed pre- and posttests and compare the resulting percentiles through the use 
of an equal interval scale called a Normal Ciirve Equivalent (NCE) which has a 
mean of 50 and a standard deviation of 21.06j| A third test is then administered 
to the students and the resulting NCE compare'4 to those^ from the first two 
tests* The data are compared to determine, whether gains made by the students' 
during the Title I/Chapter I project; have been ^sustained. The study is what we 
call the s.ust'ained effects study. 




The Compensatory Education Department, of tHk Kentui^y Department of Education 
(DOE) requested that the LEAs submit both their pi^ans and the results of their 
study of sustained effects to the state. A form wAs developed on which to 
describe the plans for the sustained effects study. , It included the following 
information: rationale for the study, major questions to be answered, the grade 
levels to be included, the name, edition, fortds and levels of the test used, and 
the dates of testing. A second form was developed on which to report the 
results of the study and the LEA's interpretation -and utilization of, the results. 

One hundred and thirty studies were reviewed and analyzed. THe purpose 
of the review was to answer the following questions: (1) What were the major 
questions of the studies? „(2) For each common question, what were the** Results? 
(3) How did the LEAS interpret the results? (4) How did the LEAfe plan to utilize 
the results? and (5) Were there any obvious tnista'kefe in the studies? The 
studies were grouped by questions asked, grade", and type of project. 

RESULT OF THE REVIEW 

What Were the fta jor Questions of. the Studies? 

V 

The major question by. far, with 109 LEAs asking, it, was: Are £he gains 
from Title I m instruction sustained over the summer? These summer effects 
studies relate'd to various combinations of grades and subjects. 

Tt\e second most frequently asked question (12 LEAS) related to exit 
from*Title I services. The questions varied as to grade level at exit and 
reasons for exit (termination of services by LEA or attainment of achievement 
level by student). , , ^ 



Other questions related to, a comparison of students in different situations 
(longer versus shorter periods of Title I instruction, summer school versus no 
summer school, team teaching versus one teacher, and three years in Titl^e I 
versus two years versus one year). 



- 7 



\ 



-2- 

Wfcat Were the Results of The Summer Effects Studies? 

— ' ; ■ . * 

The^ sample sizes varied from 3 to 421 with most in the range from 20 to 30. 
By taking a weighted (by number pf students tested) average across LEAs<for Total 
Reading* and Total Mathematics in each grade (Tables 1 and 2) a pattern for the 
state can be seen. - 4 \ 

' f TABLE 1 

SUSTAINED EFFECTS IN READING 



Grade 


•Students' 


Pretest! Posttest 


SES 
«a 


'Post - Pre 


SES - Pre 


1 VSJ^ o 

SES - Post 


r" 


12 


* 26.8 


48.9 


32.6 


22.1 


5.8 . 


-16.3 


2 ' 


695 


29.8 


37.5 


30.9 


7.7 


1.1 


- '6.6 


3 


2407 


26.5 


30.5 


28.8* 


4.0 


2.3, 


-i? 


4 


1765 


25^3 


31.8 


29s 0 


6.5 


' 3.7 


- 2.8 


5 


1110 


26.6 


32.4 


29.5 


5.8 


2.9 


- 2.9 


6 , 


588 


27.6 


33.7* 


'32.7 


6.1 


,5.1 


-. 1.0 


7 


170 


- 25.8 


29.8' 


28.9 


■ 4.0 


3.1 


- 0.9 


8 


28 


20.7 ' 


21.7 ' 


22.4 


1.0 


1.2 • 


+ 0.77 


SES ■ 


Sustained Effects Study 
< 


or the 


third 


data point 
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Grade 
2 • 
3 
*4 
5 
6 
7 

8 ' 



Number of 
Students 

62 

'360 

29C 

0 

297 
J.84 
■57 
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.SUSTAINED EFFECTS IN MAI 

Average NCEs 
Pretest . Posttest SES 



VTICS • . 

Differences in Avg. NCeV " 
Post - Pre SES - Pre SES - Post 



'26.6 
26.9 
25.1 
26.2 
28.5 
29.6 
20.4 



3419 

32.5' 

37.7 

31.7 

37.1 

32.9 

21.8 



29.4 
28.9 
32.5 
25.6 
33.1 
31.7 
29.1 



8.3 
5-. 6 
12.6 
5.5 
8.6 
3.3 
1.4 



2.8 
2.0 
7. -4' 



-5.5 
-3.6 
-5.2 



-0.6 
4.6 
2.1 
8.7 



-6.0 
-4.0 
-1.2 * . 
+7.3 
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SES - Sustained Ef fects Study ^or* the Xhird data point 

( 

• • i .'4 



There is no cbnsepsus on v the meaning of "the gains are sustained." # 
Two interpretations are (1) th£ gain from the posttest to the tl^ird 'data 
point is greater than or equal to zero, and (2) the magnitude of the loss 
from the posttest to the thirds data point is less than the gain -from the 
pre- to posttest. The second interpretation is used in this paper. 

■* . 

Overall, the above two tables show that the gains of Title I were | 
sustained. With 'the exception of 5th grade mathematics" there were NCE gains 
from pretest to the third data point. , * . * ' ' 

With the exception of 8th grade (where the sample size£ were small) there * ,* 
were NCE losses over the summer. These losses were greater in mathematics " 
Shan in reading in grades three through seven. These losses, however, may 
reflect no more Chan the difference between gains made by projects on a Fall- . 
Spring testing cycle and those made by projects on a Spring-Spring cycle. 

/ \ ^ 

How Did the LEAs Interpret the Results ? 

The LEAs 1 interpretations of the results reflected^ higher level* of 
evaluation that^ofes beyond looking at gains in test scores. They expressed 
concern^&bou* small sample sizes, lack of randomness of samples, changes "in the 
levels pf the. test, match in conterit of«the .test with objectives of the Curriculum, 
and difficulty of 'thef test. 

In many cases students who were tested at the third data point were only * 
those who were' still in a Title I program. Administrators failed to test those 
who had made greater progress and were therefor!^ 5 back in regular classes., The 
result/tfas d non-random sample of lower ability students. This was the most 
common errtfr that was found in the studies*. > „ / 

In most situations the pre- and posttests were^ of the same level while 

le thiijd test was one level higher. While the scale score system should make 

such a change equitable, there were cases — especially in the lower grades— in * 

wWich a subtest found in one level was not a part of an adjacent level. 

« 

How Did the LEAs Plan to Utilize the Results^ 
- * * 

While many LEAs were necessarily cautious in making decisions based on only % * 
one set of scores or based on a small sample of students, there were some LEAs 
that suggested ways in which they would use the results of .their sustained 
effects study. 

4>lan sumpier activities . The ne^ed for, constant reinforcement of skills 
throughout the summer was noted. Some. tf<y?ed a definite need for a summer school 
which funding would not permit. Many LEAs plan to stress .the need for summer , 
reading to parents and stifdents. (Math reinforcement seemed to be more of a * 
problem.). In some LEAs, the teachers ajre preparing materials for summer activi- 
ties that students <^an do at home. Soffie LEAs are coordinating efforts through a 
local library or bookmobile. Some are training parents in ways to help* rein- 
force reading and math skills' during the summer. Some aire giving children old 
books and other reading materials for summer use and encouraging them to use the 
librar^ during the ^summer. « ' 



Look more closely at curriculum and standardized tests * Some LEAs without 
sustained gains *are looking at their curriculum to see what needs to be added 
or deleted* Many are looking mOre closely at the'tes.ts. Some are comparing 
objectives of the test to their^j&ri curriculum and may select another test in 
the future. Some are using the test in diagnostic warys to determine in which 
areas the Students showed' the most decline fowajrd pretest NCEs. Some are 
stressing individualization and noting specific kkill deficiencies. / Some will 
plan more review in the fall prior to testing; ' \ 

Improve the placement' of children with learning difficulties . . There Was 
some concern expressed about the placement of children with learning disabili- 
ties in Title I elapses. Often students with learning disabilities and low 
achievement are placed /in Title •! classes until they c^n be identified through 
psychological testing as learning disabled. One LEA mentioned that the results 
of its st*dy had prompted the school to have more psychological testing* so that 
students l could be placed in the most appropriate learning environment. The LEA 
will alsb try to get more ckasses fir exceptional children added. 

. - • v • : - • • 

Do additional studies . The study prbmp/ted some LEAs to do more studies. 
Some indicated an interest in following ^th^/students 1 progress for more "than a 
year or in following them beyond Title I and into the high schools. One LEA 
plans to study more grades next year. One LEA planned to interview each student 

•to. see if there were patterns differentiating those who sustained their gains 
from- those who did not* Questions which they plan to ask are related to atti- 
tudes toward school study habits, outside 'activities, interests or hobbies, home 
life, and relationships with family and friends. One LEA commented that Title I 
patterns in sustained gains should be compared .with those of all children in the 

, schctfil. 

Change their Title I programs * As a» result of their studies two XEAs 
. decided to .extend their reading programs an extra year. One plans to extend t*he 
Title I program 'through a parents' activities' program. Since their best reten- 
tion was. at the early grades, one LEA toill concentrate its efforts t^ige in the 
future. Since rentention is so important, one LEA decided to allow a student # 
once selected to continue in the. Title I program until* an established goal is 
obtained with consistency. LEAs whose average gains Jjiad been -sustained usually 
were encouraged to continue the same type program. 



EDUCATIONAL SIGNIFICANCE , 

This analysis presents benchmark results by which td compare future studies 
of sustained effects over the 'summer. - It presents some ojf the problem? encoun- 
tered by LEAs and suggests, guidelines to use in the futuxe to improve the 
studies. 

The significance of thfe sustained effects* studies was not so much in the 
results of- the studies as it was in tfi£ process of- the studies. LEAs took 
a much. deeper look, at wh^t they were doing in their projects and the consequences 
of their efforts. They took a closer look at their process\of testing .and their 
interpretation of, test results in the evaluation of program impact. 
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